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ABSTRACT 

This study evaluated a Columbus (Ohio) Public Schools 
summer remedial reading and language mechanics skills program for 
participants in a job training program. The Summer Academic Skills 
Enhancement Program provided clients with the skills required for 
employment into entry-level positions in 160 curriculum hours. 
Seventy-one clients were pretested with the Comprehensive Test of 
Basic Skills (CTBS) and based on these results, were classified as 
regular pretest achievers or low pretest achievers. Forty-six black 
and 3 non-black clients (27 males and 22 females) who attended at 
least 30 days of summer program instruction and had pretest and 
posttest CTBS scores were studied. Evaluation at the end of the 
program found that pupils in the regular group reached skill 
achievement goals, while pupils in the low group did not. Program 
retention showed a 2 percent gain from the previous year (1991), and 
black male retention (82 percent) was up Ik percent from the previous 
year. Non-minority recruitment, enrollment, and retention showed no 
improvement. With program completion defined as attending 75 percent 
of the possible program days and by showing a gain of 1 or more grade 
equivalents on at least 1 of the 3 subtests, 33 clients (46 percent 
of the 71 enrolled) achieved program completion status. Analysis of 
the completion criteria suggests that requirements distort the actual 
achievement of participants. Fourteen tables display data from the 
evaluation. An appendix presents the Pupil Census Form and eight 
additional tables. (JB) 
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Abstract 



Program Description ; The Summer Academic Skills Enhancement Program (SASEP) 
was funded by the Private Industry Council (PIC) of Franklin County through 
the Job Training Partnership Act ( JTPA) . The purpose of the program was to 
provide JTPA clients with the reading comprehension and language mechanics 
skills required for employment into entry-level positions. A total of 160 
curriculum hours were used toward this end. This "Fast Track" program was 
conducted by the Department of Community Education (DCE), Columbus Public 
Schools . 

Program enrollment was defined by the Private Industry Council. 
Anticipated were 100 PIC-referred clients. A total of 71 referred clients 
v;ere pretested with the Comprehensive Test of Basic Skills (Form U, Level H) . 
These clients were enrolled into one or both of two remedial courses-of-study : 
Reading Comprehension (N=54; Houghton-Mifflin New Directions in Reading 
curriculum) or Language Mechanics (N=55; Houghton-Mifflin "whole language" 
grammar and composition series) . 

The 1992 program consisted of two segments: testing (May-June) and 
remediation (June-August). Performance objectives v;ere stated for each of the 
two remediation programs: reading comprehension, language mechanics. 

Evaluation Design ; 

1 . Evaluation Objectives for "Regular Pretest Achievers" RPA 

1.1 45% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest reading comprehension 
grade equivalent score between 5.0 and one less -than their age-grade 
placement, will show 1.0 grade equivalent gain on the posttest. 

1.2 45% of the clients enrolled in the language component who attend 75% 
of the program days and who have a CTBS pretest language mechanics 
grade equivalent score between 5.0 and one less than their age-grade 
placement, will show 1.0 grade equivalent gain on the posttest. 

1.3 70% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of 5.0 or more in their component 
area and who score below 75% on the pretest employment skills test , 
will score at or above 75% on the employment skills posttest. 

*Age-Grade Placement is the grade the client would be in without retention 
in grade. 
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2 . Evaluation Objectives for "Low Pretest Achievers" _ILPA1 



2.1 40% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest reading comprehension 
grade equivalent score less than 5 . 0 will show 1 , 0 grade equivalent 
gain on the posttest. 

? 40% of the clients enrolled Ln the language component who attend 75% 
of the program days and who have a CTBS pretest language mechanics 
grade equivalent score less than 5.0, will show 1.0 grade equivalent 
gain on the posttest. 

2.3 40% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of less than 5.0 in their 
component area and who score below 7 5% on the pretest employment 
skills test, will score at or above 75% on the employment skills 
posttest . 

3 . Other Evaluation Objectives 

3-1 75% of the clients enrolled will attend 75% of the program days. 

3.2 80% of the male/black clients enrolled in the program will be retained 
through the posttesting phase of the project, 

3.3 The clients served by the program will reflect the target population 
such that 25% of the clients will be non-minority. 

« 

Major Findings ! Pupils in the Regular Pretest Achievers (RPA) subset reached 
criterion level on both Evaluation Objectives (1.1, 1.2). To be in the RPA 
subset, a pupil had to have a CTBS pretest GE of at least 5.0. The employment 
skills test (Objective 1.3) was not administered in 1992. 

Pupils in the Low Pretest Achievers (LPA) subset failed to reach criterion 
level on either Evaluation Objective (2.1 and 2,2). To be in the LPA subset a 
pupil had to have a CTBS pretest GE less than 5.0. Particularly, only 18% 
(N=2 of 11) of LPA clients in the reading content-area, gained at least 1 . 0 GE 
at posttesting. In contrast, 39% of these LPA clients did meet the 2.2 
language criterion level (40% was required to "reach" this criterion). The 
employment skills test (Objective 2.3) was not administered in 1992. 

Program retention demonstrated a 2% gain from 1991 (69%, up from 67%; 75% 
was required). Male/Black retention (82%), up 24% from 1991, was an 
encouraging statistic. Non minority recruitment /enrollment/retention showed 
no improvement over 1991. 

A pupil attains program completion status by attending 7 5% of the possible 
program day and by showing a gain of one or more grade equivalents (GE) on at 
least one of the three CTBS subtests. In this context, "ceiling effect" 
continues to confound assessment of achievement gain and to depress the number 
of clients who achieve program-completion status. As was articulated in the 
1991 final report, numerous clients scored at or above 12.0 (grade equivalent 
on at least one of the three CTBS (UH) tests used to evaluate pretest-to- 
posttest change. Such a pretest score defeats the program's purpose on both 
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counts (gain measurement and program completions). See recommendation #3 
below. 

Thirty-three clients achieved "program completion" status. This group 
represents about 46% of the 71 pupils enrolled in the 1992 Summer Program. 
Data collected during evaluation of program clients suggest that th^ number of 
completions could be increased significantly by reducing the attendance 
requirement from 75% (30 days) to about 60% (say, 23 days). Or, base 
completion solely on whether a pupil completes all pre- and postte5?t3. Either 
way, the 1992 Summer Program v/ould have had at the least 8-lQ more 
"completions." This is significant because what is being propof'Sd is an 
additional 10-15% more completions. Pupils enrolled for less thc^n 30 days, 
even with perfect attendance, cannot — by definition — become program 
completions, although they may well gain one or more GE's on one a- more of 
the three tests. 

"Enrollment," especially calculati.on of a percent attending "75% of the 
instructional days offered, is influenced by a large number of pupils who take 
the CTBS pretest and then do nothing else. While it may be correct to use the 
number of clients enrolled as the denominator for calculating ^'retention" 
data, it appears probable that (under prevailing selection criteria) the 
clientele served by this program is unlikely ever to reach an 80^ retention 
level. An alternative algorithm is worth investigating; or, a lower criterion 
value might be considered. 

Recommendations 

1. There does not seem to be a compelling reason for lowering criteri-on levels 
for assessment of the 1993 SASEP. 

2. Special emphasis should be given to examination of the procegs used to 
expose LPA clients to the reading program. 

3. It is recommended that a new definition of "completion" be derived and that 
this definition be applied in 1993, in each instance where higfi (>=12.0) 
pretest CTBS GE values so warrant. 

4. It is recommended that program sponsors and managers test the viability of 
using a "completion" definition in the future which is either* (a) not 
dependent on attendance, (b) is relaxed to, say, 60%, a level that would 
have captured the six pupils lost to attendance in 1992 v/ho ciid gain at 
least 1.0 GE, and (c) allows for pupils enrolled for less than the full 39 
days of instruction. 

5. Continued emphasis on timely enrollment and daily attendance should 
increase the proportion of 1993 SASEP clients who attain 30 ancj more days 
of attf^ndance. 

6. Retention of male/black clienta should be given the highest priority for 
1993. 

7. Program planners for the 1993 SASEP should structure and implement 
effective, productive strategies for attracting, enrolling anci retaining 
non-minority clients . 
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FAST TRACK 

FINAL EVALUATION REPORT 
SUMMER AC7VDEMIC SKILLS ENHANCEMENT PROGRAM 

1992 

Program Description 

The 1992 FAST TRACK Summer Program consisted of two distinct phases: 
testing and remediation. The testing phase was designed to identify pre-test 
performance levels; remediation strategies were adopted to maximize the 
potential for improving clients' content mastery in two instructional areas: 
reading comprehension (six objectives); (b) language mechanics (five 
objectives). The proposed Program Design statement summarized these two 
phases as "Testing" and "Remediation". Underlying the remediation phase was a 
singular goal: provide JTPA clients with the academic skills necessary for 
"employment into entry level positions-" 

Employment Skills," an integral part of past PIC Summer Programs, was not 
offered in 1992. 

Referral and Selection 

The target group for this program was defined as "JTPA eligible youths aged 
14-21" . 

The selection process was initiated by a referral from the Private Industry 
Council (PIC). Referrals were tested and the Columbus Public Schools •( CPS ) 
Department of Community Education (DCE) notified PIC regarding performance 
levels . 

An anticipated 100 PlC-identif ied youth were to be pretested. All PIC 
referrals were enrolled to attend the eight-week remediation phase beginning 
in mid-June. The remediation phase was conducted at the North Education 
Center and emphasized prescriptive/individualized instructional strategies and 
materials. 

Recruitment Methods : The Private Industry Council supplied (by way of the PIC- 
10 referral form) the Department of Community Education with the names of 71 
eligible youth who were chosen or self-identified for participation in the 
Summer Academic Skills Enhancement Program (SASEP) . in April, the Department 
administered a CTBS battery to those youth. All 71 clients were selected by 
PIC for the "Fast Track" program. The department also assisted PIC officials 
with orientation and enrollment. 

Testing 

Commencing May 1, 1992 and through June 26, 1992, the Department of 
Community Education administered the Comprehensive Tests of Basic Skills 
(CTBS; 1981 ) , Form U, Level H reading comprehension, language mechanics, and 
mathematics computation subtests to 71 PIC-identif led youth. The principal 
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assessment activity for students enrolled in the Columbus City School District 
took place at students* home schools. For youth living outside the Columbus 
City School District or for youth referred after the testing period was 
closed, testing was administered by PIC personnel at a site designated by PIC 
management. The Departm.^nt of Community Education supplied PIC staff with the 
test instruments and answer sheets. 

The Department of Program Evaluation (DPE) of the Columbus Public Schools 
scored completed answer forms and produced individual diagnostic reports and 
system summaries. All scores were norm-ref erenced» The Columbus evaluators 
used the TESTMATE microcomputer software system to scan, score, and report 
norm-referenced data. 

The Department of Community Education, in concert with the Department of 
Program Evaluation selected Form U, Level H of the Comprehensive Tests of 
Basic Skills (third edition) as the most appropriate level of difficulty for 
the identified client group. The CTBS is a norm-referenced achievement test, 
the content categories of which were defined by excimining current state and 
district curriculum guides, published texts and instructional programs, and 
criterion-referenced assessment instruments. Columbus evaluation 

professionals selected the reading comprehension, language mechanics, and 
mathematics computation subtests for administration to clients. Total time 
for actual testing was 93 minutes; test administration protocols added 
approximately 35 minutes to the testing session. 

Reading . At the lowest levels, the reading comprehension test measures visual 
and sound recognition of letters, words, vowels, and consonants. Items 
measuring comprehension skills are related to sentences and stories. Reading 
comprehension items measure skills in understanding sentence meaning, passage 
details, character analysis, main ideas, generalization, written forms, and 
author techniques. (Houghton-Mifflin's New Directions in Reading) 

Language Mechanics . These items measure the student's ability to identify the 
correct use of capital letters, periods, commas, exclamation points, question 
marks, quotation marks, colons, semicolons in sentences and in extended 
passages. (Houghton-Mifflin's "whole language" grammar and composition series) 

Testing Methodology Used . The tests' designers used a three-parameter Item 
Response Theory (IRT) to scale the CTBS and to develop norms • Application of 
IRT methodology provides a number of direct benefits to the user of CTBS Form 
U, including more acjurate descriptions of client performance. Consultants 
from the educational community, represented by native American, Asian, 
Hispanic, and Black ethnic and cultural groups, reviewed all items for 
possible racial, ethnic, and gender bias. Consecjuently , the standardized 
instruments do not contain items that appeared statistically biased in item 
tryouts. In the standardization, the sample reflects ethnic minorities as 
they are represented in the general population* 

Remediation 

Seventy-one PIC-referred youth were enrolled in an e ght-week summer 
prescriptive and individualized instructional program at the North Education 
Center as part of the Summer Youth Employment Training Program operated by the 
Private Industry Council* The instructional phase of the remedial program 
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took place from June 5, 1992 through August 7, 1992. Tho»e clients who 
successfully completed course work were eligible to receive 0.5 unit of 
academic credit for reading, or 0.5 unit of academic credic for language arts. 

Clients attended daily classes in reading comprehension and/or language 
arts. Client instructional hours began at 8:10 a,m. and concluded at 11:50 
a.m., Monday through Friday. All training was delivered by instructors 
certificated by the State of Ohio. 

At the conclusion of the summer instructional phase, sta'cf administered the 
CTBS (UH) to clients retainer? to that point. The Department of Program 
Evaluation analyzed data using .^c'P^op^iate descriptive statistics to determine 
whether the summer remedial treatment was effective in improving clients* 
basic academic skills. 

The Department of Community Education chose instructional materials based 
on research findings that have correlated student learning with patterns of 
curriculurr organization. Specifically, researchers discovered that highly 
s?:ructured instructional formats are most effective when working on basic 
skills comp'/^tencies with lower achieving students. The following curricula 
were designed to achieve maximum mastery over a short time through rigorous 
instructional organization: 

Reading Comprehension. The curriculum employed was Houghton-Mifflin's New 
D irections in Reading program, which has been designed as a reading 
comprehension achievement aeries for high school students who have not yet 
mastered reading comprehension skills. The three-part instructional plan 
consists of (a) preparation in vocabulary building, (b) enhancement of 
compreherision skills through guided reading, and (c) review and extension 
exerciser, to verify comprehension and provide skills reinforcement through 
immediate practice. The comprehension domain is the central focus of each 
instructional unit, and the curriculum stresses 10 comprehension skills: 
understanding punctuation, understanding word referents, using context to 
reveal word meanings, and to understand figurative language / noting important 
details, understanding sequence of events, recognizing the main idea of 
paragraphs, making inferences and drawing conclusions or predicting outcomes, 
understanding cause-effect relationships, understanding comparisons, and 
distinguishing between fact and opinion. In addition to quizzes for 
individual lessons, instructors administered both mid-level and end~of -level 
testing. 

Language Mechanics. The language curriculum used Houghton-Mifflin's "whole 
language" grammar and composition series. This curriculum integrates grammar 
with reading and writing skills. Grammar units begin with the presentation of 
the basic lesson, and from that base progresses to vocabulary building 
activities. These activities are capped by exercises that assist students to 
make the crucial grammar-writing connection. Students then move to ''checkup" 
activities that assess mastery levels attained-. A cumulative review follows, 
which in turn io supplemented by enrichment work or differentiated additional 
practice (easy, average, or challenging). Reading and writing units commence 
with literature selections and are followed by activities that give students 
practice in using the three modalities of literature response: listening, 
speaking, and thinking (inferring/drawing conclusions). Composition skills 
are taught through the five-step writing process: pre-writing, drafting. 



P:\P552\P1CFRT92 
11-18-92 11:11 AM 



ERIC 



4 



revising, proofreading, and publishing (final drafting). Students master 
spelling skills using ?Ioughton-Mif f lin ' a spelling prograni, which supports a 
complete testing program in standardized test format. 



Evaluation Design 

Pretesting of program candidates was used to discern skills deficiencies in 
reading, language, and mathematics. Then, clients were enrolled into either 
or both of two programs: Reading Comprehension or Language Mechanics. 
Program clients were then guided through the remediation phase as described 
above. Clients who completed the eight -week instructional program were then 
posttested to reveal pre/post test change with respect to reading 
comprehension, language mechanics, and mathematics computation observed 
scores . 

Because the Summer 1991 Program evaluation design resolved the so-called 
"student ID number problem" successfully, this year care again was taken to 
standardize the number-assignment process. The Department of Program 
Evaluation (DPE) pre-printed CTBS answer forma and prepared lists of students 
who were pretested for use by the Summer Program Coordinator. Summer 1992 
program personnel used these lists to code student numbers on all forms 
returned to the DPE for analysis. Thus, it again was possible this year to 
conduct analysis as intended, analyses based on attendance, where 30 days (75% 
of 39 enrollments days) of attendance was the cut-off point. 

Completion 

Two project-completion criteria were evaluated. To be considered a 
"completion," a client must: 

1. attend 75% (N==30) of the 39 enrollment days; and 

2. demonstrate a grade-equivalent gain (pretest to posttest) of at least 
1.0 on any one of the three CTBS subtests: Reading Comprehension, 
Language Mechanics, or Mathematics Computation. 

Evaluation Objectives 

Nine evaluation objectives were stated in the DCE/CPS 1992 Summer Program 
proposal. These objectives were initiated by the DPE, following review of the 
1992 program. It was suggested that these nine objectives accurately 
represent the program's potential for measuring clients* CTBS scores. But, 
expectations for improving CTBS scores should be tempered by an awareness of 
the wide range of clients* pretest scores and the likelihood of improvement 
therein . 

Thus, evaluation objectives were stated for two groups: Regular Pretest 
Achievers (RPA; pretest grade-equivalent scores of 5.0 or more). Low Pretest 
Achievers (LPA; pretest grade-equivalent scores of less than 5.0). Also, 
objectives regarding Reading Comprehension and Language Mechanics were 
structured for accommodation of "age-grade placement" (the grade a client 
would be in without having been retained) . 
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RPA/LPA Rationale . 

an SASEP is a remediation-type, ghort-term learning experience. Of 
interest to program planners is whether strategies adopted to guide the 
instructional process work equally well with both RPA and LPA groups. 

Prior to 1992, SASEP data analysis did not include provisions for 
assessment of these two entry-level-split groups. In 1991, following 
examination of 1990 SASEP data the decision to analyze test results with 
respect to RPA and LPA membership was proposed for the 1992 SASEP. This 
analysis plan and the results of its implementation were documented in the 
1991 final report submitted to PIC. Because that analysis did reveal some 
insights useful for program-planning purposes, the same analysis model was 
used in 1992. 

The "5.0 GE" demarcation was arbitrary. It was chosen because observation 
of 1990 data seemed to indicate that "at around this point" would separate 
v;hat appeared to be two groups of clients with divergent CTflS pretest scores 
in reading and language; i.e., frequency distributions of pretest scores were 
bimodal at approximately GE = 5.0. 

And, program planners wanted to krow how well the Reading Comprehension 
program was working with respect to CTBS-measured pre/posttest reading gain; 
how well the Language Mechanics program was working with respect to 
pre/posttest language gain. Therefore, each group (RPA/LPA) was assigned one 
reading (1.1, 2.1) and one language objective (1.2, 2.2). 

Age-Grade Placement 

Selection into the RPA group was calculated to include pupils who scored 
higher on the CTBS pretest than those entered into the LPA group, but lower by 
at least 1.0 GE than students performing at grade-level. Thus, a client in 
the RPA group was known to be performing lower than would be expected 
routinely. The rationale used for structuring selection in this way was based 
on the not remarkable fact that these clients should be excellent targets for 
realizing the 1 . 0 GE gain on one of the three CTBS subtests. 

So, a 9th-grade client scoring a 9.0 GE for Reading Comprehension would be 
at "age-grade." Conversely, a 9th-grade client scoring 8.0 GE for Reading 
Comprehension would be 1.0 behind age-grade. For Evaluation Objectives 2.1- 
2.3, analyses were based on clients scoring 1.0 or more lower than their 
respective age-grade placements. 

1. Evaluation Objectives for "Regular Pretest Achievers" (RPA) 

1.1 45% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest reading comprehension 
grade equivalent score between 5.0 and one less than their age-grade 
placement*, will show 1.0 grade equivalent gain on the posttest. 

1.2 45% of the clients enrolled in the language component who attend 75% 
of the program days and who have a CTBS pretest language mechanics 
grade equivalent score between 5.0 and one less than their age-grade 
placement*, will show 1.0 grade equivalent gain on the posttest. 
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* Age-Grade Placement ia the grade the client would be in without retention 



1.3 70% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of 5.0 or more in their component 
area and who score below 75% on the pretest employment skills test, 
will score at or above 75% on the employment skills posttest. 

2 . Evaluation Objectives for Low Pretest Achievers (LPA) 

2.1 40% of the clients enrolled in the reading component who attend 75% of 
the program days and v/ho have a CTBS pretest reading comprehension 
grade equivalent score less than 5.0, will show 1.0 grade equivalent 
gain on the posttest* 

2.2 40% of the clients enrolled in the language component who attend 75% 
of the program days and who have a CTBS pretest language mechanics 
grade equivalent score less than 5.0, will show 1.0 grade equivalent 
gain on the posttest. 

2.3 40% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of less than 5.0 in their 
component area and who score below 7 5% on the pretest emp.loyment 
skills test, will score at or above 75% on the employment skills 
posttest . 

2 . Other Evaluation Objectives 

3.1 75% of the clients enrolled will attend 75% of the program days. 

3.2 80% of the male/black clients enrolled in the program will be retained 
through the posttesting phase of the project (excluding clients 
expelled) . 

3.3 The clients served by the program will reflect the target population 
such that 25% of the clients will be non-minority. 

Methodology 

Seventy-one prospective clients were pretested with the CTBS. Fifty-eight 
CTBS posttest forms were scored. Demographic and attendance data were 
recorded on revised Pupil Census Forms (PCF) for all youth pretested with the 
CTBS. File folder reports for all 71 youth pretested were computer-generated 
by the Department ot Program Evaluation. These reports were customized to 
include demographic/attendance data and the results of each test taken by the 
pupil, and to record program-completion status. See the Appendix for examples 
of the reports provided by DPE. 

Achievement data were scanned and scored using TESTMATE computer software. 
Demographic and attendance data were encoded by this consultant. Preliminary 
reports to program administration were provided each time new data were added 
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to the dataaetfl. These analyQGS were conducted using an IBM 9375 running CMS. 
Thi& hardware and SAS 6.07 were used to analyze CTBS data in terms of grade- 
equivalent change. 

At CTBS pretesting (N=71), 33 (46%) were female and 38 (54%) were male. 
Regarding ethnicity, 5 (7%) were non-minority and 66 (93%) black. No Asians 
or American Indians were among the 71 clients in 1992 (see Appendix). 

At CTBS posttesting, 51 clients had at least 30 days of registered 
attendance. Two of these 51 did not have posttest scores reported for 
analysis. The remaining 49 clients were split 22 female (45%), 27 male (55%), 
approximating pretest gender percentg. 

The Evaluation Sample 

To be included in the evaluation sample, a client must have attended at 
least 30 days of Summer Program Instr'jction. Since "completion" requires 75% 
attendance v 30 of 39 enrollment days), given that achievement gain is-- 
clearly — a function of being exposed to elements of a formal instructional 
process, it was determined thac the evaluation sample--appropriateiy — would 
include only those clients who met this criterion. 

For analysis, it was recpjiired that a client meeting the attendance 
criterion also must have pre- and posttest CTBS scores (at least one subtest 
pair). Therefore, only potential "program completions" were included in the 
analysis. A total of 49 clients met both evaluation sample eligibility 
criteria* 



The 49 clients (see Appendix) analyzed as the evaluation sainple were 
predominantly black, split about 45%, 55% female/male. A total of 46 black 
clients were analyzed; only three non-black clients are in the evaluation 
sample (all non-minority). As a group these three clients represent just G% 
of the evaluation sample. 

Separate analyses of CTBS data were conducted for each of the evaluation 
criteria stated earlier. Because the evaluation sample — essentially — is black 
by ethnic group, furth^^r analyses using this variable — ethnicity— were not 
justified . 

Program Completion Analysis 

To be designated a program "completion," a client must: (a) attend 75% (30 
days) of the enrollment period; and (b) show a grade-equivalent gain of at 
least 1.0 on one or more subtests (pre/poettest) of the CTBS. The evaluation 
sample included 49 clients with data sufficient to teat for program 
completion. (A client missing the CTBS posttest could not become a 
"completion," by PIC definition). 

This completion-candidate group of 49 clients consisted of 22 (45%) 
females, 27 (55%) males, 46 (94%) blacks? of the black group 21 (46%) were 
female and 25 (54%) were males. Thirty-six were enrolled in Reading 
Comprehension, and 38 in Language Mechanics. Eleven were enrolled in reading 
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only; 13 were enrolled in language only; and 25 were enrolled in both reading 
and language. Thus, about 67% (36 of 54) of those enrolled in Reading 
Comprehension were completion candidates. Of the 38 enrolled in Language 
Mechanics, 69% (38 of 55) were completion candidates. Twenty-five (51%) of 
the 49 completion clients were enrolled in both Reading Comprehension and 
Language Mechanxcs. 

Thirty-three (67% of the evaluation sample) clients achieved completion 
status, 16 females and 17 males. Thirty-one of these 33 were blacks Fifteen 
(94%) of black/female completion candidates did reach program completion 
status; 16 (94%) of black/males were completions. 

The 33 clients who achieved program-completion status were enrolled in one 
of three content^area classifications: Reading Comprehension only, Language 
Mechanics only, received instruction in both reading and language. Six (18% 
of total completions) received only Reading Comprehension instruction; nine 
(27%) received only Language Mechanics instruction; and 18 (55%) received 
instruction in both content areas. 

Evaluation Sample enrollment was not equal across the three instruction 
types: 11 in Reading Comprehension, 13 in Language Mechanics, and 25 in both. 
Proportionately, six (55%) of the 11 reading clients achieved program 
completion; nine (69%) of the 13 language clients achieved completion; and 18 
(72%) of the 25 in the "both" group achieved program completion. 

A total of 47 "G.E.>=1.0" gains were observed for clients across the three 
content-area types: reading = 17 (36* of 49), language = 21 (45%) and math = 9 
(19%). 

Forty-nine clients were program-completion candidates (> 30 days of 
attendance and valid posttest minus pretest score). Thirty-three (67%) of the 
49 achieved program completion. Among the 49 candidates, 47 difference scores 
>=1.0 were observed; in fact, all 47 difference-score "completions" were 
observed among the 33 clients who achieved program-completion status . In 
summary: eight (24% of 33) gained >=1.0 GE in Reading Comprehension (RC); nine 
(27%) in Language Mechanics (LW) four (12%) .n Mathematics Computation (MC) ; 
seven (21%) RC + LM; zero (0%) RC + MC; three (9%) LM + MC: and two (6%) RC + 
LM + MC. 

How does a gain >=1.0 in, for example, reading relate to language and math? 
Consider Table 1, where RCG is Reading Comprehension gain, LMG is Language 
Mechanics gain. RCG, LMG, and MCG = 1 if gain >=1.0. This table also 
includes the variable "COMPL" which is a "1" if program completion was 
achieved. The correlation between reading and language gain >=1.0 is 0.15, 
and the likelihood of paring program completion w- ^h RCG is 0.51; prograim 
completion and LMG correlate 0 . 60 . None of the RCG/LMG/MCG correlations 
suggest strong relationships. Thus, gain >=1.0 in any one content area 
appears — essentially — unrelated to that in either of the other two areas. 

The narrative in the remainder of the report references Tables 1-14, The 
following variable names are used in these tables. 
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Variable Name 


Comments 


ATTEND 


Number of days of attendance - 


ATT 


Attend 30 days = YES, NO otherwise. 


PROG 


Instructional curriculum • 


ETHNIC 


Ethnic group. 


GERCA 


Grade equivalent, Reading Comprehension pretest. 


GERCB 


Grade equivalent, Reading Comprehension posttest. 


GELMA 


Grade equivalent, Language Mechanics pretest. 


GELMB 


Grade equivalent. Language Mechanics posttest. 


GEMCA 


Grade equivalent. Math Computation pretest. 


GEMCB 


Grade equivalent. Math Computation posttest. 


AGEGR 


Age-Grade . 


AGEGRPLA 


Age-Grade placement. 


RCG 


Reading Comprehension gain; 




>1.0 GE = YES, NO otherwise. 


LMG 


Language Mechanics gain; 




>1.0 GE = Yes, NO otherwise. 


MCG 


Math Computation gain; 




>1.0 Gj; = YES, NO otherwise. 


SEX 


Gender ; M=male , F=f emale 


COMPL 


Program Completion l=Yes, NO otherwise 



Table 2 presents pertinent frequencies for the 71 FAST TRACK clients who 
constituted the "potential" completions group. Table 3 presents both pre- and 
posttest descriptive statistics for the "Base Means", where base is a term 
used to identify all 71 Fast Track clients as a group. Next, (see Tablo 4) 
the same variable frequencies displayed tor the "non-completion" subsets. 
And, Table 5 presents similar frequencies for the "completion" subset. Table 
6 shows means and other descriptive statistics useful for comparison with the 
base-means in Table 

Regarding data in Taible 6, note the almost seven-day difference in 
c.ttendance. On average. Fast Track clients who completed the 1992 Summer 
Program missed less than three of the 39 instructional days. Non-completion 
clients missed nearly 10 days. 

Interestingly, on average, pretest scores in all three content areas were 
higher in the non-completion group than in the completion group. However, 
pretGst-to-posttest "gain" (defined in terms of the observed change in grade- 
equivalent scores measured), is almost zero for members of the non-completion 
group. The Reading Comprehension (0.52) and Language Mechanics (0.64) "gains" 
for the completion group are above 0.50. Gain in Math Computation was 0.27 
for the completion group. Clearly, transfer-if any — of skills acquired in 
reading and/or language may have had something to do with the 0.27 gain 
observed in Math Computation by members of the completion group. 

Table 7 displays pertinent frequencies for members of the evaluation sample 
(N=49). The saimple has only three non-black members (no Asians or American 
Indians). Forty-six of the 49 program completions are black; 21 (46%) are 
female and 25 (54%) are male. The common factor, of course, is 30 or more 
days attendance by all members of the evaluation sample. Moreover, 33 (67%) 
of the 49 members of the evaluation sample gained at least 1.0 GE on at least 
one of the three tests. 
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Evaluation Objectives for "Regular Pretest Achievera" RPA 

1.1 45% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest reading comprehension grade 
equivalent score between 5-0 and one less than their age-grade placement, 
will show 1.0 grade equivalent gain on the posttest. 

A total of 13 clients were in this group. Each of the 13 had an age- 
grade placement exactly one lees than age-grade. These are clients who 
were enrolled in the Reading Comprehension content area and whose pretest 
score suggested one year of retention. 

Ten of the 13 possible completions (77%) from this group were realized. 
Six of the 13 (46%) were Reading Comprehension completions (see Table 8), 
clients who gained at least 1.0 GE from pretest to posttest. 

This criterion was reached. 

1.2 45% of the clients enrolled in the language component who attend 75% of 
the program days and who have a CTBS pretest language mechanics grade 
equivalent score between 5.0 and one less than their age-grade placement, 
will show 1.0 grade equivalent gain on the posttest. 

This objective is nearly identical to E.G. 1-1, the only exception being 
that the content area Language Mechanics is of interest. Four clients 
enrolled in Language Mechanics constitute this group. Of these four, 
three (75%) v/ere program completions. Tv/o of the four (50%) were 
Language Mechanics completions (see Table 9). 

This criterion was reached. 

1.3 70% of the clients who attend 75% of the program days and who have a CTBS 
pretest grade equivalent score of 5.0 or more in their component area and 
who score below 75% on the pretest employment skills test, will score at 
or above 75% on the employment skills posttest. 

The 1992 Fast Track Summer Program did not include Employment Skills 
instruction . E . O . 1.3 was not evaluated . 

Evaluation Objectives for "Low Pretest Achievers" (LPA) 

2 . 1 40% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest reading comprehension grade 
equivalent score less than 5.0 will show 1.0 grade equivalent gain on the 
posttest . 

Eleven LPA clients enrolled in the Reading Comprehension content are also 
had reading pretest-scores of less than 5.0 GE* Of these 11, eight (73%) 
were program completions. But, only two of the eight completions (25%) 
were associated with a gain of 1.0 GE or more pretest-to-posttest in 
Reading Comprehension. Thus, two of 11 (18%) met this criterion see 
Table 10) . ^ 
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This criterion was not reached. 

2.2 40% of the clients enrolled in the language component who attend 75% of 
the program days and who have a CTBS pretest language mechanics grade 
equivalent score less than 5.0/ will show 1,0 grade equivalent gain on 
the posttest. 

Twenty-eight LPA clients from the evaluation sample (57%) were enrolled 
in the Language Mechanics content area, all of whom had a language 
pretest score of less than 5.0. Eighteen (64%) were program completions, 
11 (39%) of whom were completions in Language Mechanics (see Table 11). 

This criterion was not reached. 

2.3 40% of the clients who attend 75% of the program days and who have a CTBS 
pretest grade equivalent score of less than 5.0 in their component area 
and who score below 75% on the pretest employment skills test, will score 
at or above 75% on the employment skills posttest* 

The 1992 Fast Track Summer Program did not include Employment Skills 
instruction. E*0. 2.3 was not evaluated. 



Other Evaluation Objectives 

3.1 75% of the clients enrolled will attend 75% of the program days. 

Of the 71 clients enrolled in the 1992 Fast Track Summer Program, 49 
(69%) attended at least the minimum of 30 days needed toward program 
completion. Although 69% represents a 2% gain from 1991, the criterion 
was not reached (see Table 12). 

This criterion was not reached, 

3.2 80% of the male/black clients enrolled in the program will be retained 
through the posttesting phase of the project. 

Thirty-five male/blacks enrolled in one or more program content areas and 
were pretested. Of these thirty-five, only eight (23%) were not 
posttested, two of whom were expelled. A rather striking retention rate 
"gain" of 24 percentage points was realized from 1991 to 1992 (see Table 
13) . 

This criterion was reached. 

3.3 The clients served by the program will reflect the target population such 
that 25% of the clients will be non-minority. 

Only five non-minority (white) clients were enrolled and only three of 49 
( 6% ) were in the evaluation sample. One of the non-minority non- 
completions was not posttested and the other attended just 29 days (see 
Table 14) . 

This criterion was not reached.. 
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Summary /Recommendat iona 

Nine evaluation objectives (E.O.) were proposed for assessment of the 1992 
Summer Academic Skills Enhancement Progr2Lm( SASEP ) . Two of these (1,3, 2.3) 
were dependent upon measurement of Employment-Skills items. Because the 1992 
program did not include instruction (or measurement) in the Employment Skills 
area, summer-program assessment was based on the remaining seven E.O.'s. 

These seven E.o.'s — for the most part — were identical to those used for 
assessment of the 1991 SASEP; comparison of results from 1991 to 1992 is 
possible. The only difference between E.O.'s across years is seen in the 
percent needed to reach criterion. These differences are the result of 
applying one of the 1991 program-planning recommendations in that final 
report, "...rethinking criterion levels appropriate for 1992 is recommended." 
(p. 11) Rethought criterion levels were submitted in DCE's proposal. 

Both "Regular Pretest Achievers" (RPA) objectives were reached. Neither of 
the "Low Pretest Achievers" (LPA) objectives were reached. For reasons 
unknown to this consultant, LPA Reading (18%) fell far below the criterion 
(40%). But LPA language (39%) was only one percent below criterion (40%). 
RPA reading was up 9% (37% to 46%, 1991-92). RPA language was up 10% (40% to 
50%). LPA reading was down 18% (36% to 18%). LPA language was down 61% (100% 
to 39%). (It is important to note that in 1991 there were only three LPA 
clients in the Language Mechanics group.) 

There does not seem to be a compelling reason for lowering 
criterion levels for assessment of the 1993 SASEP . 

With the limited information available (N=ll) , it ia difficult to identify 
the cause of so few LPA reading-program clients reaching the GE>1.0 criterion 
(N=2). Of course, these clients enter the program by-definition less prepared 
than members of the RPA group. Nevertheless, LPA members offer — by and large 
— a better opportunity for growth, and knowledgeable, intensive programming 
should improve the proportion of LPA clients who do become program 
completions. Is there a systematic reason why LPA language clients (39%) 
should show more than double the percent of content-area completions? 

Special emphasis should be given to examination of the process 
used to expose LPA clients to the reading program . 

The 1991 SASEP final report succinctly delineated recommendations regarding 

(a) definition of "completion" and (b) the number of days of actual 
attendance . These recommendations were offered because completion/ attendance 
standards in effect during the 1991 SASEP were shown to detract from valid 
assessment of that summer program: (a) pupils scoring 12.0 or higher on a 
pretest could not possibly gain a full grade equivalent pretest-to-posttest ; 

(b) pupils attending, say, 29 days of instruction could and did show GE gain 
greater than 1.0 but were eliminated for lack of one day of attendance. 

Both situations appeared, again, in 1992. The recommendations offered in 
1991 are repeated again this year. If the goal is to gain at least 1.0 GE in 
a content area, fine, but — for a client enrolled only 30 days — expectation of 
perfect attendance probably is too high, realistically* 
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One thing is certain: "completions," per se . under the current mandates, 
consistently will underestimate positive gains made by SASEPs. 

It is recommended that a new definition of "completion" be der ived 
and that this definition be applied in 1993, in each instance 
where high (>=12.0) pretest CTBS GE values so warrant . 

It is recommended that program sponsors ancjl managers test the 
viability of using a "completion" definition in the future which 
is either (a) not dependent on attendance, (b) is relaxed to> sav^ 
60%/ a level that would have captured the six pupils lost to 
attendance in 1992 who did gain at least 1.0 GE, and (c) allows 
for pupils enrolled for less than the full 39 days of instruction . 

Seventy-two percent (51) of the 71 Fast Track clients in 1992 attended at 
least 30 days of formal instruction. This represents a 4% increase from 1991. 
Client attendance beginning with day one is important* 

Continued emphasis on timely enrollment and daily attendance 
should increase the proportion of 1993 SASEP clients who attain 30 
or more days of attendance . 

A major gain (24%) was made this year in terms of the number of black/male 
enrollments who were retained through posttesting. Clearly, progress was made 
in 1992. Since over one-half of all enrollments were black/male, it is worthy 
to reiterate increased emphasis on retention of this particular group. 

Retention of male/black clients should be given the highest 
priority for 1993 . 

Obviously — and regrettably—enrollment of non-minority clients (N=5; 7%) 
for the 1992 SASEP was far below criterion (25%). This 7% represents only a 
2% gain from 1991. Recruitment strategies used to at:tract non-minority clients 
to PIC summer programs have not been reviewed by this consultant, so no 
comment should be presented in this context. it is not apparent why non- 
minority representation should continue to fall far below reasonable 
enrollment expectations. The 1991 recommendation needs to be repeated. 

Program pl anners for the 1993 SASEP should structure and implement 

effective, productive strategies for attracting, enrolling an^ 

retaining non-minority clients . 
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Table 1 

Correlation of Program Completion and Difference-Score Gain 
(All Variables are Dichotomous* ) 



COMPL RCG LMG MCG 

RCG 0.51 1.00 0.15 -0.12 

LMG 0.60 0.15 1.00 0.12 

MCG 0.33 -0.12 0.12 1.00 

* N=49 for each variable 



Prepared by 
Office of the Deputy Superintendent 
Department uf Program Evaluation 
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PIC 1992 SUMMER PROGRAM 
PUPIL CENSUS FORM 



PIC Student Name 

PIC Student Number 

CPS Student Number 

Sex F Grade ^ Birth Date 061477 



PIC Room /^/ 



Ethnic Group Membership (circle one) 



1 - Non~Mi nor i ty 



'7}}- Black 



Total Days of Program Attendance ^ ^ Days. 3 - Spanish Surname 4 - Asian 
Total Days o£ Program Enrollment 33 Days . 5 - Amer ican Indian 



INSTRUCTIONS FOR COMPLETING FORM 



1. Enter PIC room number. 

2. Enter Columbus Public School student number. 

3. Enter grade (grade last year). 

4. Verify preprinted values (name, PIC student number, sex, and birth date). 

(Write in changes above preprinted values.) 

5. Circle the appropriate ethnic group. 

6. Enter total days of attendance by this pupil. 

7. Enter total days this pupil was enrolled in the Summer Program this year, 

(Note: days of enrollment must be greater than or equal to days of 
at tendance . ) 

8. In the graphic below, use an "X" (as appropriate) to indicate both 
"Program" and "Instructional-Content Area" for this pupil; e.g., a 
pupil in the STEP program receiving Language Mechanics instruction 
should have an "X" in the row STEP, under the column heading Language 
Mechan i cs . 

Use an "X" to indicate each content area in which this pupil receives 
instruction. An "X" may appear in more than one column but in only 
one row. 



CONTENT AREA 



CCh'T^tl. corJ-nwvi-t.'Z- <^^A;r^f-3 

Reading Language Mathematics 

— PROGRAM Comprehension Mechanics Computation 



// -7 /' 
J) 



FAST 


TRACK ^ 






STEP 








FAST 


TRACK 2 
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PIC Student Name 
PIC Student Numbar 
CPS Student Number 
Sex F Grade 08 



PIC 1992 SUMMER PROGRAM 
PUPIL CENSUS FORM 

(Fi le Folder Copy) 
PIC Room 



121 



Birth Date 06 1477 



Total Days of Program Attendance 28 . 0 Days. 
Total Days of Program Enrollment 39 . 0 Days. 



ETHNIC GROUP MEMBERSHIP (circled) 
1 - Non-Minority Q - Black 

3 - Spanish Surname 4 - Asian 
5 - American Indian 



C ONTENT AREA 

Reading Language Mathematics 

PROGRAM Comprehens 1 on Mechan i cs Computa 1 1 on 

Fast Track X x 

Step 

Fast Track II 



C T B S TESTING RESULTS 

(Grade Equivalent Values) 

Pretest Post test Change 

Reading Comprehension 6.5 9.3 2.B 

Language Mechanics 4.0 5.3 1.3 

Math Computation 5.0 4.6 -0.4 



••••• COMPLETION STATUS •--«« 

• Cr 1 ter 1 on Status 
« 

• Attendance (30 Day Min.) NO 
« 

• CTBS Grade-Equivalent Gain (GE>-1.0) YES 

• Program Completion NO 



Prepared by 
Office of the Deputy Superintendent 
Department of Program Evaluation 

Run Date: 09SEP92 
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